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Abstract 

The  military  training  community  strives  to  meet  ever-increasing  expectations  and  challenges.  One  expectation  is  to 
support  all  levels  of  training  from  novice  to  expert.  Supporting  the  full  spectrum  of  training  curricula,  from  basic 
knowledge  acquisition  in  a  classroom,  or  e-learning,  to  on-the-job-training  with  embedded  training  and  mission 
rehearsal  capabilities,  are  all  of  great  importance  to  maintaining  a  qualified  Fleet.  Challenges  to  the  delivery  of 
training  include:  1)  increased  operational  tempo  and  mission  complexity,  2)  the  perishable  nature  of  higher  order 
cognitive  skills  and  team  behaviours,  3)  reduced  availability  of  instructors  and  role-players,  4)  system  deployment 
and  usability,  and  5)  costs  including  travel  and  leave  costs  of  personnel  while  at  “brick  and  mortar”  schoolhouses 
(Lyons,  Schmorrow,  Colin.  &  Lackey,  2002).  An  approach  to  addressing  theses  challenges  is  to  apply  advanced 
artificially  intelligent  computer  assisted  instruction  technology,  also  known  as  Intelligent  Tutoring  Systems  (ITS) 
(Chipman  2005),  throughout  the  training  continuum  of  delivery  mechanisms.  We  propose  that  additional 
improvements  could  be  realized  across  the  training  continuum  by  creating  Augmented  ITS  capabilities  that  combine 
state-of-the-art  ITS  components  with  augmented  cognition  technologies,  such  as  cognitive  and  neurophysiological 
sensing,  and  warfighter  assessment  gauges.  This  technology  is  currently  being  developed  under  DARPA’s 
Improving  Warfighter  Information  Intake  Under  Stress  Program,  also  referred  to  as  Augmented  Cognition  or 
AugCog.  The  resulting  Augmented  ITS  will  add  a  new  level  of  capability  to  blended  training  solutions. 

This  paper  will  provide  an  introduction  and  overview  into  the  newly  immerging  field  of  Augmented  Cognition  for 
Training  Superiority  and  Education.  It  is  being  presented  as  the  1st  of  seven  papers  in  a  session  of  the  1st 
International  Conference  on  Augmented  Cognition.  Throughout  the  paper  we  propose  approaches  for  inserting  data 
from  cognitive  and  neurophysiological  sensing,  and  warfighter  assessment  gauges  into  the  student  model,  expert 
model  and  tutor  components  of  intelligent  tutoring  systems  to  create  Augmented  ITS.  Brief  summaries  of  on-going 
research  in  this  area  being  conducted  by  our  colleagues  are  provided  for  illustration.  Notional  examples  of 
Augmented  ITS  applied  to  both  individual  and  team  training  are  also  offered.  Finally,  we  conclude  with  expected 
benefits  towards  addressing  the  challenges  to  military  training  and  recommendations  for  future  research. 

1  Introduction 

The  military  training  community  strives  to  meet  ever-increasing  expectations  and  challenges.  One  expectation  is  to 
support  all  levels  of  training  from  novice  to  expert.  Supporting  the  full  spectrum  of  training  curricula  (Figure  1), 
from  basic  knowledge  acquisition  in  a  classroom,  or  e-learning  setting,  to  on-the-job-training  with  embedded 
training  and  mission  rehearsal  capabilities,  are  all  of  great  importance  to  maintaining  a  qualified  Fleet. 


Training  Continuum 

Schoolhouse/Lecture/Lab  practice  --  e-learning/ADL  —  Virtual/M&S  —  Live  --  Embedded  —On-the-job 

Figure  1:  The  continuum  of  training  delivery  mechanisms  utilized  by  the  military. 

Challenges  to  the  delivery  of  training  include:  1)  increased  operational  tempo  and  mission  complexity,  2)  the 
perishable  nature  of  higher  order  cognitive  skills  and  team  behaviours,  3)  reduced  availability  of  instructors  and 
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role-players,  4)  system  deployment  and  usability,  and  5)  costs  including  travel  and  leave  costs  of  personnel  while  at 
“brick  and  mortar”  schoolhouses  (Lyons,  Schmorrow,  Cohn,  &  Lackey,  2002).  An  approach  to  addressing  theses 
challenges  is  to  apply  advanced  artificially  intelligent  computer  assisted  instruction  technology,  also  known  as 
Intelligent  Tutoring  Systems  (ITS)  (Chipman  2005),  throughout  the  training  continuum  of  delivery  mechanisms. 
We  propose  that  improvements  could  be  realized  across  the  training  continuum  by  combining  Augmented  Cognition 
Technologies  such  as  cognitive  and  neurophysiological  sensing,  and  warfighter  assessment  gauges  with  state-of-the- 
art  ITS,  providing  Augmented  ITS  capabilities. 

The  Navy  has  established  several  commands  to  address  education  and  training.  The  Naval  Education  and  Training 
Command  (NETC)  including  the  Human  Performance  Center  (HPC)  and  Navy  Personnel  Development  Command 
(NPDC)  focus  on  the  initial  portion  of  the  training  continuum  (individual  training),  with  an  innovative  initiative 
called  the  Integrated  Learning  Environment  (ILE).  Fleet  Forces  Command  addresses  the  other  end  of  the  continuum 
with  collective  Fleet  Training  once  a  sailor  or  marine  is  assigned,  or  deployed,  with  modeling  and  simulation  (M&S) 
based,  distributed  training  exercises  that  can  include  deployable,  embedded  and  live  training  platforms.  Success  in 
maintaining  qualified  personnel  throughout  their  careers  is  contingent  on  continuous  collaboration  between  these 
organizations. 

The  goal  of  the  Integrated  Learning  Environment  (ILE)  is  to  improve  and  support  job  performance  and  mission 
readiness  by  providing  high  quality  learning  and  performance  support  available  anytime  and  anywhere.  This  will  be 
accomplished  by  providing  a  single  system  for  sailors  to  view  and  plan  their  entire  career  and  advancement  process. 
Sailors  will  be  using  elements  such  as  the  Navy  Knowledge  Online  website,  the  Five  Vector  Model,  and  an 
individualized  electronic  training  jacket  (ETJ)  tool  to  track  their  credentials  and  requirements  for  advancement 
(Human  Performance  Center,  2004). 

An  essential  element  of  the  ILE  is  migrating  relevant  areas  of  its  classroom  learning  to  web-based  learning 
combined  with  modeling  and  simulation  practicum  experiences  represented  by  “simulation  objects”  which  could  be 
enhanced  with  augmented  ITS  capabilities.  This  migration  is  expected  to  reduce  the  time  and  cost  to  educate  and 
train  sailors  while  increasing  operational  readiness  by: 

•  Enabling  prescriptive  learning  through  reusable  learning  objects  that  can  be  combined  "on  the  fly"  to 
deliver  individualized  instructional  curriculum,  referred  to  as  "my  course." 

•  Increasing  individual  and  mission  performance  by  making  knowledge  available  to  sailors  in  the  fleet  when 
and  where  it  is  needed. 

•  Providing  training  according  to  a  "the  right  sailor  with  the  right  skills  in  the  right  job  at  the  right  time  for 
the  right  cost"  approach. 

•  Emphasize  learner  control  and  responsibility. 

In  order  to  maximize  the  success  and  payoffs  of  the  entire  training  continuum  including  those  used  by  the  ILE  and 
M&S  based  training  approaches,  it  is  imperative  that  we  understand  and  optimize  the  employment  of  these  delivery 
systems.  Such  optimization  can  only  truly  be  achieved  by  matching  the  individual’s  needs  for  learning  to  the  time 
spent  in  training  focused  on  targeted  training  objectives.  It  is  this  need  to  understand  the  individual’s  state  of 
knowledge,  skills  and  abilities,  which  we  believe  can  be  improved  by  the  addition  of  augmented  cognition 
technologies  in  training. 

2  Background  on  Intelligent  Tutoring  Systems 
Scope  and  Definition 

The  development  of  ITS  began  in  the  1970s  as  an  attempt  was  made  to  enhance  computer  based  instruction  (CBI).  . 
The  goal  was,  and  still  is,  to  combine  highly  individualized  CBI  with  artificial  intelligence  (AI)  methods  in  an 
attempt  to  produce  the  same  kind  of  flexibility  and  learning  effect  that  occur  when  a  human  tutor  and  student  work 
together  (Siedel  and  Park,  1994).  In  meeting  this  goal,  the  ITS  would  have  the  capability  to  assess  what  the  student 
already  knows,  the  curriculum  that  student  needs  to  learn,  and  the  appropriate  instructional  strategy  and  feedback 
format  to  be  used.  These  decisions  are  made  continuously  and  in  real  time.  The  flexibility  of  the  system  to  make 
these  decisions  is  what  distinguishes  ITS  from  CBI.  Seidel  and  Park  (1994)  cite  four  features  that  are  intrinsic  to 
ITS  that  are  impossible  to  include  in  CBI: 


1.  ITS  can  generate  traditional  knowledge,  rather  that  select  pre-programmed  frames  containing 
knowledge  to  present  to  the  student. 

2.  ITS  allow  both  the  system  and  student  to  initiate  instructional  activities  by  applying  AI  techniques 
used  to  develop  the  computers  natural  language  dialogue  capability. 

3.  ITS  can  make  inferences  in  interpreting  the  student’s  inputs,  diagnosing  misconceptions  and  learning 
needs,  and  generating  instructional  presentations. 

4.  ITS  can  monitor,  evaluate,  and  improve  its  own  tutoring  performance  by  applying  AI  techniques 
commonly  used  in  machine  learning. 

Components  of  ITS 

By  definition,  an  ITS  must  contain  three  essential  components:  the  expert  model,  the  student  model,  and  the  tutor 
(Figure  2).  We,  and  our  colleagues  in  this  Augmented  Cognition  for  Training  Superiority  and  Education  Session, 
propose  various  methods  to  “augment”  all  three  components  of  ITS.  The  expert  model  consists  of  knowledge  of  the 
subject  domain  and  is  the  basis  for  comparison  of  information  received  from  the  student  model.  Although  all 
components  of  the  ITS  are  integral  to  its  ability  to  provide  instruction,  Anderson  (1988)  considers  the  expert  model 
to  be  the  backbone  to  any  ITS,  as  the  system  could  not  function  without  domain  knowledge.  There  are  multiple 
options  for  encoding  information  into  the  expert  domain.  A  black  box  model  involves  finding  a  way  of  reasoning 
about  the  domain  that  does  not  require  codifying  the  knowledge  that  underlies  human  intelligence.  Another  option 
involves  developing  an  expert  system  and  requires  the  human  expert  knowledge  to  be  codified  and  applied  to  the 
ITS.  A  more  complex,  yet  powerful  method  involves  incorporating  a  cognitive  model,  which  authors  such  as 
Anderson  (1988)  and  Zachary  (1999)  suggest  is  essential  to  producing  high  performance  tutoring  systems.  We 
propose  that  representing  cognitive  state  or  warfighter  assessment  data  (i.e.  working  memory  load  and  attention)  in 
the  expert  models  of  ITS,  particularly  when  using  cognitive  models,  will  provide  a  deeper  level  of  comparison  and 
diagnosis  of  the  student’s  knowledge  with  potential  improvement  in  training  effectiveness  of  the  tutoring. 


Artificially  Intelligent  Computer  Assisted  Instruction 
aka  Intelligent  Tutoring  Systems  (ITS) 


— -  AugCog  Technologies  Insertion 

Figure  2:  Intelligent  Tutoring  Systems  and  proposed  “Augmented  ITS”  approaches. 

(Adapted from  Cannon-Bowers  and  Lyons  (a.k.a.  Nicholson),  1999) 


The  student  model  represents  the  current  state  of  the  learners’  knowledge.  The  ITS  assesses  what  the  learner  already 
knows  during  the  diagnostic  phase.  During  this  phase,  the  ITS  typically  uncovers  the  student’s  cognitive  state  from 
his  or  her  observable  behavior  (Van  Lehn,  1988).  There  are  several  diagnostic  techniques  that  are  currently  being 
used  to  assess  the  student  model.  The  model  tracing  technique  is  the  easiest  and  most  widely  used.  This  technique 
assumes  that  the  diagnostic  program  has  access  to  all  of  the  student’s  significant  mental  states.  This  approach  works 


by  “delineating  many  hundreds  of  production  rules  that  model  curricular  chunks  of  cognitive  skill.  A  learner’s 
acquisition  of  these  chunks  and  departure  from  the  optimal  route  is  immediately  remediated”  (Shute  and  Postka, 
1996).  There  are  many  other  techniques  used  to  construct  the  diagnostic  module  that  is  built  in  to  the  student  model. 
They  include  path  finding,  condition  induction,  plan  recognition  and  issue  tracing.  We  propose  that  combining  the 
trainee’s  cognitive  state  and  neurophysiological  data  with  the  typical  observable  behavioural  data  represented  in  the 
student  model  will  result  in  improved  diagnosis  and  improved  tutoring  results. 

The  last  component  of  the  ITS  is  the  tutor.  This  component  encompasses  the  knowledge  of  teaching  strategies  to  be 
employed  during  the  learning  process  (Halff,  1988).  After  a  trainee’s  knowledge  has  been  assessed  and  compared  to 
the  domain  knowledge  in  the  expert  model,  the  tutor  must  then  make  decisions  regarding  curriculum  and 
instructional  method,  in  order  to  bridge  the  knowledge  gap.  Although  there  are  many  mitigation  strategies  that 
could  be  employed,  leading  approaches  include  the  delivery  of  real-time,  on-line  feedback,  adapting  the  curriculum 
or  scenario  and  post  training  debriefing  in  the  form  of  a  summary  of  performance.  We  propose  that  the  selection 
and  specific  implementation  of  these  strategies  can  be  informed  by  the  data  provided  through  the  augmented 
cognition  technologies  to  effectively  map  the  mitigation  strategy  to  the  student.  For  instance,  a  novice  student  may 
require  more  explicit,  amplifying  information  in  an  on-line  feedback  message  where  as  a  more  expert  trainee  could 
benefit  from  a  simple  notification  of  a  mistake.  Another  area  that  could  benefit  from  the  sensing  technologies  is 
maintaining  the  trainee’s  interest  or  level  of  engagement.  A  foundational  principle  of  the  Augmented  Cognition 
program  is  that  based  upon  the  Yerks  and  Dodson  inverted  U  curve  of  human  performance,  there  exists  an  optimum 
balance  between  human  performance  and  a  person’s  level  of  stress  or  arousal  (Mendl,  1999).  One  goal  of  the 
adaptive  human  computer  interface  manipulations  used  by  the  Augmented  Cognition  teams  is  to  maintain  the 
operator  at  that  peak  of  performance.  We  propose  that  the  same  theory  could  be  used  to  drive  adaptation  of  a 
training  scenario  or  level  of  difficulty  of  the  curriculum,  striving  to  maintain  a  student  at  a  peak  level  of  learning. 

We  include  a  review  of  our  colleagues’  specific  research  efforts  exploring  several  implementations  of  these  concepts 
for  augmenting  ITS  in  section  4  below.  However,  we  will  first  provide  an  overview  of  the  augmented  cognition 
technologies  that  we  keep  referring  to  from  DARPA’s  Improving  Warfighter  Information  Intake  Under  Stress 
Program,  also  referred  to  as  Augmented  Cognition  or  AugCog. 

3  Augmented  Cognition  Technologies 

New  and  established  neurophysiological  sensor  technologies,  coupled  with  sophisticated  cognitive  modeling 
software,  have  the  potential  to  provide  additional  capabilities  to  traditional  ITS  approaches  employed  in  existing  and 
emerging  human-centric  training  solutions  such  as  scenario-based  training  in  virtual  environments  (Lackey,  2004)  or 
web-based  training.  An  impressive  collection  of  cognitive  state  detection  and  physiological  sensor  technologies  and 
tools  are  currently  under  investigation  by  the  AugCog  researchers  (see  Table  1).  These  tools  are  being  investigated 
to  determine  which  individual  technology  or  combination  of  technologies  contributes  to  the  development  of 
Warfighter  Assessment  Gauges.  Though  initially  being  applied  to  adaptive  human  computer  interaction  (HCI) 
configurations  for  operational  domains,  applying  these  Warfighter  Assessment  Gauges  to  training  environments  will 
allow  for  experimentation  to  determine  the  level  of  contribution  each  cognitive  state  detection  tool  and  physiological 
measure  can  provide  within  an  augmented  ITS. 


Cognitive  State  Detection  &  Physiological  Sensor  Technologies 

Cortical  blood  oxygenation 

Interbeat  Interval 

Cortical  blood  volume 

Respiration  Rate 

Event  related  optical  signal 

Galvanic  Skin  Response 

Neuronal  patterns 

Posture 

Neuronal  firing  signatures 

Eye  fixation  duration 

Frequency  of  neuronal  population  firing 

Eye  gaze  location 

Sync/Desync  of  neuronal  structures 

Pupil  dilation 

Error  related  negativity 

Mouse  Pressure 

P300 

Rate  of  task  completion 

Heart  Rate 

Error  rate 

Table  1:  Detection  and  sensor  technologies  of  interest  to  AugCog  (Schmorrow  et  al.,  2005) 


The  AugCog  teams  are  each  working  with  various  combinations  of  the  above  sensors  to  develop  Warfighter 
Assessment  Gauges  for: 

■  Attention, 

■  Verbal  Working  Memory, 

■  Spatial  Working  Memory, 

■  Memory  Encoding, 

■  Error  Detection,  and 

■  Autonomic  Activity 

The  gauges  are  based  upon  data  output  streams  from  combinations  of  sensors,  simultaneously  captured  from  an 
individual  in  real-time.  The  resulting  capabilities  have  resulted  in  the  ability  to  address  the  four  cognitive 
bottlenecks  depicted  in  Figure  3.  Sample  mitigation  strategies  follow  in  Table  2  and  are  paired  with  task 
environments  investigated  by  the  four  AugCog  research  teams. 


Input  Cognitive  Output 


Figure  3:  Cognitive  bottlenecks  (Schmorrow,  et  al.,  2004) 


Cognitive  Processing 
Bottlenecks 

Sample  Task  Mitigation  Strategy 

Task  Environment 

Attention 

Task  scheduling  strategy  based  on  priority,  time  and 
operator  readiness 

Dismounted  Soldier 

Working  Memory 

Alert  delivery  strategy  based  on  verbal  vs.  spatial 
workload  of  operator 

Aegis  Command  and 
Control  Supervisor 

Executive  Function 

Bookmark  cueing  strategy  based  on  independent  or 
dependent  task  context  switching 

Unmanned  Vehicle 
Controller 

Sensory  Input 

“Sensory  shortcut”  strategy  based  on  available  input 
channels 

Command  and  Control 
Vehicle  Operator 

Table  2:  Processing  bottlenecks  addressed  by  AugCog 

AugCog  efforts  to  mitigate  cognitive  bottlenecks  have  provided  compelling  tools  to  improve  Warfighter 
performance  with  computer  assisted  adaptive  HC1  in  these  operational  environments.  The  expectation,  based  on  this 
success,  is  that  these  technologies  will  also  support  the  improved  implementation  of  computer  assisted  instruction  by 
improving  the  effectiveness  of  training  mitigations  such  as  on-line  feedback,  adaptive  scenarios  or  level  of  difficulty 
in  curriculum  and  post  training  after  action  review. 

4  On-going  research  in  Augmented  ITS 

This  paper  is  striving  to  provide  an  introduction  and  overview  into  the  newly  immerging  field  of  Augmented 
Cognition  for  Training  Superiority  and  Education.  It  is  being  presented  as  the  1st  of  seven  papers  in  a  session  of  the 
1st  International  Conference  on  Augmented  Cognition.  The  following  is  a  brief  summary  of  each  of  our  colleagues’ 
research  approaches  towards  achieving  the  Augmented  ITS  capability  described  above.  The  first  summary  of 
Campbell  and  Luu  describes  an  approach  for  augmenting  the  Student  Model  of  Figure  2.  The  remaining  researchers 
provide  various  approaches  or  examples  of  augmenting  the  Tutor  Module  of  the  ITS. 


Detection  of  Slips  versus  Mistakes  (Campbell  &  Luu,  2005) 

Campbell  and  Luu  are  performing  research,  under  ONR’s  Virtual  Technologies  and  Environments  (VIRTE) 
Program,  to  investigate  the  feasibility  of  using  neurophysiological  indicators,  particularly  error-related  negativity 
(ERN),  when  interpreted  in  conjunction  with  performance  data,  to  support  improved  automated  diagnosis  in  a 
simulation-based  ITS.  Their  concept  is  that  ERN  signals  have  been  used  to  identify  when  a  person  recognizes 
immediately  that  he  or  she  has  made  an  error.  This  situation  corresponds  to  a  behavioural  “slip”  in  Norma’s  (1981) 
discussion  of  error  types,  and  does  not  reflect  a  true  mistake  indicating  an  underlying  misconception  or  knowledge 
gap,  which  would  require  the  same  type  or  amount  of  instructional  remediation.  They  propose  to  first  use  the  ERN 
response  slip  data  to  filter  the  student  performance  data  set  being  feed  into  a  student  model  to  increase  the  amount  of 
variance  accounted  for  by  a  mathematical  model  of  performance  data.  Once  this  is  demonstrated,  the  next  step  is  to 
evaluate  the  impact  of  incorporating  this  neurophysiological  analysis  into  an  augmented  ITS  in  a  simulation  based 
training  system. 

Detection  of  transition  from  Novice  to  Expert  (Cohn,  Kruse,  &  Stripling,  2005) 

The  goal  of  most  training  and  educational  sessions  is  to  move  a  novice’s  behaviours  and  patterns  of  knowledge  and 
reasoning  towards  those  of  the  expert’s.  As  discussed  previously,  typical  ITS  assess  the  trainee  at  the 
outcome/behavioural  level.  Another  team  of  scientists  under  the  VIRTE  Program  are  investigating  the  hypothesis 
that  there  are  detectable,  reliable  and  repeatable  behavioural  and  neuro-physiological  measures  that  distinguish 
experts  from  novices  in  complex  training  tasks.  They  propose  the  use  of  continuous  collective  performance 
measures,  or  a  “Performance  Template”,  that  incorporates  both  behavioural  and  neuro-physiological  measures. 
Through  the  use  of  these  collective  measures  the  goal  is  to  gain  the  capability  to  track  a  trainee,  in  real  time,  through 
the  stages  from  novice  to  expert  using  technologies  sensitive  to  the  dynamic  nature  of  learning  in  the  brain.  If 
successful,  an  augmented  ITS  can  be  realized  in  which  it  will  be  possible  to  assess  where  a  trainee  is  cognitively  in 
the  training  process  and  to  then  move  trainees  more  efficiently  and  effectively  through  the  training  curriculum. 

Mitigating  Barriers  to  Sympathetic  Teaching  and  Testing  (DuRousseau,  2005) 

This  effort  focuses  on  the  barriers  to  sympathetic  teaching  and  testing,  and  addressing  those  barriers  by  developing 
and  using  a  Learning  System  Developer’s  Kit  (LDSK).  DuRousseau  describes  the  process  for  combining  cognitive 
state  assessment  algorithms  with  off-the-shelf  computer-adaptive  teaching  (CAT)  software  capable  of  intelligently 
managing  the  delivery  of  instruction.  In  addition,  tools  to  facilitate  the  design  and  test  of  applications  which  are 
capable  of  manipulating  content,  timing,  style,  and  form  of  multimedia  stimuli  in  real-time  based  on  measures  of 
executive  control  within  the  human  brain  are  described.  A  summary  of  recent  work  focused  on  the  computer- 
adaptive  testing  architecture  and  analytical  methods  of  the  LDSK  is  included. 

Cognition  monitoring  for  pedagogical  diagnosis  and  intervention  (Mathan  &  Dorneich,  2005) 

Cognitive  tutors,  a.k.a.  ITS,  are  used  to  monitor  student  behaviour  dynamically,  and  using  model  tracing  techniques, 
intervene  when  student  behaviours  are  not  consistent  with  strategies  determined  to  lead  to  successful  task 
performance.  Anderson  and  Gluck  (2001)  have  noted  that  ITS  have  several  limitations  when  compared  to  live 
instructors,  specifically  with  respect  to  the  ability  to  detect  cognitive  and  affective  states.  However,  Mathan  and 
Dorneich  propose  that  the  use  of  neurophysiological  sensors  to  detect  cognitive  and  affective  states  could  be  used  to 
effectively  augment  ITS  to  enhance  the  diagnostic  process  in  the  Tutor  Module,  thereby  more  closely  linking  student 
performance  and  skill  level  with  appropriate  automated  training  interventions.  Implications  for  ITS  interventions 
can  be  more  closely  linked  to  skill  maturity  level,  producing  more  efficient  training. 

Physiologial  self-regulation  training  (Pope  &  Prinzel,  2005) 

One  challenge  of  long-duration  space  flight  is  cognitive  skill  maintenance.  Pope  and  Prinzel  have  developed 
RESTORE,  physiological  self-regulation  training  technologies  that  can  be  embedded  in  engaging/entertaining  tasks 
and  adapt  the  task  scenario  based  on  physiological  signals  derived  via  EEG  to  modulate  parameters  of  the  task  to 
optimize  skill  maintenance.  These  scenario-adapting  approaches  are  proposed  to  be  applied  to  other  operational  and 
training  domains,  such  as  submarine  operations,  to  maintain  and  reinforce  cognitive  readiness  and  monitor  and 
maintain  effective  mood  states. 

Potential  predictors  of  errors  in  shooting  judgment  and  cognitive  performance  by  law  enforcement  personnel  during 
a  stressful  training  scenario.  (Meyerhoff,  Saviolakis,  Burge,  Norris,  Wollert,  Atkins,  Speilberger,  and  Hanson,  2005) 
Significant  military  research  has  centered  on  performance  during  stressful  situations.  This  work  adds  to  this  body  of 
literature  by  investigating  the  relationship  between  stress  induced  elevated  arousal  levels  and  performance.  The 


authors  describe  a  study  in  which  the  arousal  level  of  police  trainees  was  assessed  after  they  completed  a  highly 
stressful  interactive  scenario.  Although  stress  induced  elevated  cortisol  levels  have  been  linked  to  poor  performance 
in  a  previous  study,  the  results  of  the  current  study  did  not  follow  that  trend.  However,  the  results  did  suggest  that 
another  indicator  of  arousal,  increased  heart  rate,  appears  to  be  linked  to  performance.  The  authors  then  discuss 
whether  these  indicators  of  arousal  can  be  used  as  possible  predictive  measures  of  performance  decrement,  as  well 
as  the  development  of  measure  to  keep  arousal  levels  balanced  during  stressful  situations.  This  research  could  have 
implications  for  accounting  for  levels  of  stress  within  an  augmented  ITS. 

5  Aviation  Simulation-based  Training  System  Example  of  Augmented  ITS 

The  following  examples,  previously  mentioned  in  AugCog  International  Quarterly  (2005),  provide  insight  into  the 
potential  impact  AugCog  technologies  through  Augmented  ITS  can  have  on  individual  and  team  training. 

Individual  Training 

The  achievements  of  the  AugCog  program  make  it  possible  to  more  than  imagine,  but  to  speculate  what  future 
military  training  will  look  like  in  the  next  decade.  A  young  Lieutenant  seeking  additional  Naval  Flight  Officer 
(NFO)  training  serves  as  an  illustration.  LT  Thompson,  Air  Control  Officer  (ACO)  -  Air  Wing  5,  climbs  into  his 
NFO  E-2C  Hawkeye  training  simulator  to  practice  tracking  and  communication  skills  for  an  upcoming  qualification 
check.  As  part  of  his  pre-flight  routine,  he  dons  his  helmet  and  safety  harness.  Integrated  into  his  gear  are  various 
cognitive  and  physiological  sensors.  From  sensor  data  being  fed  into  the  student  model  and  tutor  module  of  the 
augmented  ITS,  the  NFO  simulator  automatically  alters  the  scenario  on  the  fly  to  challenge  the  ACO’s  weaknesses 
and  provide  extra  practice  events  where  needed.  LT  Thompson’s  instructor  gains  deeper  understanding  of  the 
Lieutenant’s  intentions,  actions,  workload,  and  ultimately  his  performance  in  real  time  from  the  Warfighter 
Assessment  Gauges  being  displayed  on  the  instructor’s  operator  station.  The  increased  awareness  of  the  instructor 
will  allow  for  improved  during  action  feedback  and  assessment,  as  well  as  a  more  detailed  after  action  review 
specifically  tailored  to  the  Lieutenant’s  needs.  Integration  of  AugCog  technologies  into  the  ITS  results  in  two 
valuable  improvements:  a  truly  individualize  training  experience  for  the  Lieutenant,  and  value-added  tools  that 
facilitate  an  instructor’s  ability  to  efficiently  and  effectively  enhance  individual  readiness. 

Team  Training 

An  exciting  opportunity  for  the  training  community  is  the  development  of  an  AugCog  sensor  suite  capable  of 
measuring  comprehensive  team  training  performance.  This  capability  involves  monitoring  team  members’  mental 
workload  through  the  use  of  cognitive  and  physiological  sensors.  The  ability  to  capture  such  data  provides  an 
opportunity  for  an  ITS  to  facilitate  training  of  teamwork  skills  such  as  proper  communication  and  back-up  behavior 
between  teammates.  Team  back-up  is  appropriate  when  teammates  have  the  responsibility  to  recognizes  one 
another’s  workload  and  step  in  to  cover  or  reallocate  the  distribution  of  tasks  when  feasible.  Additionally, 
identification  of  specific  scenario  events  which  contribute  to  the  onset  of  mental  overload  by  the  Warfighter 
Assessment  Gauges  can  provide  instructors  with  much  needed  insight  into  team  performance. 

Extending  the  example  above,  LT  Thompson,  an  ACO,  returns  to  his  E-2C  training  simulator  for  an  Air  Wing 
training  event  that  includes  members  of  his  immediate  team  -  Combat  Information  Center  Officer  (CICO)  and 
Radar  Officer  (RO)  and  several  F/A-18  Sweep  and  Strike  air  assets.  During  a  difficult  Dynamic  Strike  simulation 
exercise,  AugCog  sensors  track  the  workload  of  the  three  E-2C  flight  officers.  These  sensors  indicate  that  LT 
Thompson’s  workload  level  crosses  the  productive  threshold  during  the  middle  of  the  training  scenario.  At  the  same 
time,  the  sensors  indicate  that  the  RO  is  the  best  candidate  to  relieve  a  portion  of  the  ACO’s  tasking.  Rather  than 
continue  to  overwhelm  his  teammate,  as  part  of  the  established  crew  contract  and  his  expected  teamwork  skills,  the 
RO  should  assume  responsibility  for  monitoring  incoming  verbal  communications  and  hooking  new  tracks.  If  this  is 
not  observed  by  the  Team  Tutor  module,  the  ITS  will  provide  immediate  feedback  to  the  RO  notifying  him  of  his 
teamwork  responsibilities  and  recommend  that  he  provide  the  needed  back-up  to  his  teammate.  This  allows  the 
ACO  to  focus  on  providing  outgoing  verbal  and  digital  communications  to  the  Sweep  and  Strike  assets  until  sensors 
indicate  that  he  is  cognitively  prepared  to  recover  all  of  his  primary  tasking.  By  incorporating  AugCog’s  cognitive 
and  physiological  sensor  technology  for  this  training  event,  individual  and  team  performance  is  monitored  in  order 
to  train  productive  team  skills  and  improve  overall  team  performance. 
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Current  capabilities  and  future  directions 


The  promise  of  augmented  ITS  capability  will  be  of  no  consequence  if  they  fail  to  deliver  effective  training  defined 
by  a  demonstrable  improvements  in  task  performance,  skill  maintenance  or  other  relevant  training  outcome.  To 
achieve  full  potential  there  are  several  capabilities  that  must  be  effectively  designed  and  developed.  First,  trainee 
cognitive  state  must  be  effectively  assessed  in  real-time.  Several  efforts  currently  underway  under  the  DARPA 
Improving  Warfighter  Information  Intake  Under  Stress  Program  are  focusing  on  this  problem.  One  team  led  by  the 
Boeing  Company  (Barker,  2004),  for  example,  has  developed  a  closed-loop  system  for  augmenting  cognitive 
performance  of  Unmanned  Aerial  Vehicle  operators.  They  have  demonstrated  real-time  measurement  of  verbal 
working  memory,  spatial  working  memory  and  executive  function  using  a  combination  of  EEG,  pupilometry  and 
functional  near  infrared  (fNIR).  Russell  (2004)  has  developed  a  neural-net  based  EEG  signal  processor  for  this 
team  that  can  classify  EEG  activity  in  real-time,  and  which  feeds  into  cognitive  state  classification  software  that 
computes  relative  activity  levels  of  various  types  of  cognitive  activity.  This  team  has  also  developed  software  that 
modulates  the  user  interface  of  the  UAV  operator,  based  on  cognitive  state  data,  to  optimize  cognitive  processing 
throughput.  In  this  example,  the  user  interface  alternates  between  relatively  textual  (verbal)  and  graphical  (spatial) 
displays  to  take  advantage  of  relative  surpluses  in  verbal  and  spatial  working  memory  capacities.  They  have 
demonstrated  significant  improvements  in  operator  task  performance  under  augmented  conditions  (AugCog 
International  Quarterly,  2005). 

Although  the  Boeing  work  is  an  example  of  adaptive  HCI  in  an  operational  environment,  the  application  readily 
transfers  to  the  training  domain.  The  components  developed  by  this  team  are  the  same  fundamental  components  that 
would  be  required  in  an  augmented  ITS:  real-time  cognitive  state  measurement  and  classification,  software  to 
modify  the  interface  or  task  based  on  cognitive  state,  and  metrics  that  provide  assessment  of  operator  or  trainee 
performance. 

Other  related  efforts  are  underway  under  DARPA,  ONR  and  OSD  sponsorship  to  address  the  usability  and 
ruggedness  of  the  sensor  technologies  such  as  non-contact  EEG  electrodes,  wireless  electrodes,  new  optical  sensors, 
and  miniaturization  of  fNIR  systems.  These  initiatives  have  the  potential  to  provide  the  components  for  deployable, 
maintainable  and  reliable  augmented  ITS  training  systems.  However,  much  research  remains  for  the  future, 
particularly  with  respect  to  implementation  techniques  for  augmented  ITS  as  well  as  training  effectiveness 
evaluations  of  such  systems. 

7  Conclusion 

We  have  proposed  that  improvements  could  be  realized  across  the  training  continuum  from  e-learning  to  deployed 
training  by  combining  Augmented  Cognition  Technologies  such  as  cognitive  and  neurophysiological  sensing,  and 
warfighter  assessment  gauges  with  state-of-the-art  ITS,  providing  Augmented  ITS  capabilities.  It  is  our  premise  that 
the  realization  of  these  capabilities  will  be  useful  for  addressing  the  many  challenges  currently  faced  by  the  military 
training  community. 

We  predict  that  the  techniques  described  in  this  paper  will  optimize  the  effectiveness  of  ITS  training  systems  and  be 
key  enablers  for  providing  the  individualized  and  tailored  training  required  to  overcome  the  challenges  of  increased 
operational  tempo  and  mission  complexity.  Such  optimization  will  make  the  most  of  every  training  opportunity, 
providing  the  basis  for  streamlining  the  training  pipeline  in  an  attempt  to  combat  the  general  high  costs  of  training. 

Consequences  of  increasing  operational  tempos  in  today’s  military  environment  include  reductions  in  the 
availability  of  instructors,  teammates  or  other  role-players  for  training  scenarios.  For  a  sailor  deployed  aboard  ship 
for  six  months,  the  non-availability  of  instructors  or  teammates  could  result  in  significant  degradation  of  perishable 
higher-order  cognitive  skills  and  team  behaviours  if  there  were  not  tools  available  for  skill  maintenance.  The  advent 
of  augmented  ITS  coupled  with  realistic  synthetic  teammates  (Schaafstal,  2001)  in  deployable  trainers  should 
provide  significant  improvement  in  skill  maintenance  and  refresher  training  capabilities. 

The  reduced  reliance  on  live  instructors  resulting  from  the  implementation  of  augmented  ITS  provides  new 
opportunities  for  cost  savings  through  deployed  training.  However,  deployment  of  trainers  with  this  technology  will 
place  new  requirements  for  the  reliability  of  the  system  components.  The  sensor,  state  monitoring,  and  adaptive 
control  technologies  emerging  from  the  field  of  augmented  cognition  must  still  undergo  significant  development  to 


become  sufficiently  rugged,  small  and  reliable  before  such  systems  can  come  into  widespread  use.  Additionally,  to 
capitalize  on  true  cost  savings  one  must  consider  the  return  on  investment.  Certainly  there  will  be  cost  saving 
benefits  of  reduced  reliance  on  live  instruction  and  live  teammates,  both  in  terms  of  manpower  and  infrastructure. 
However,  if  these  savings  are  offset  by  significant  costs  related  to  development,  procurement  and  maintenance  of 
new  automated  augmented  training  solutions,  the  question  of  their  value  will  remain.  Nevertheless,  it  is  encouraging 
that  efforts  are  being  made,  by  programs  such  as  DARPA’s  Augmented  Cognition,  to  address  issues  (i.e., 
technology  maturation  and  deployability)  that  are  critical  to  future  training  capability,  so  that  the  time  horizon  to  the 
widespread  availability  of  affordable  and  deployable  augmented  training  systems  should  be  significantly  shortened. 
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